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ABSTRACT 



A gcaphical user interface system is {Hovided having an 
evoit-driven control program with a unified user input 
pointing event stream without distinction ci source and a 
graphical user interface having a virtual dis[day space con- 
taining objects. The control program psooessing events 
related to a plurality of user inputs representing different 
locations within the virtual display space, and opmtes 
optionally in two modes. A first mode permits the plurality 
of user inputs to interact, allowing an object in the virtual 
dis|>lay space at a first location specified by an initial user 
input to be associated with a second location specified by a 
concurrent and subsequently tenninadng user input In this 
case, the control program has a counter which increments on 
activity of any user input and decrements on cessation of 
activity of any user input, for determining an initial user 
input condition, temporal ooncuzrenoe of a plurality of user 
inputs, and a tenninal user input condition. A second mode 
fvocesses a scries of events related to a first user input 
without apparent functional interference widi a ooncunent 
scries of events related to a second user input. 

32 Claims, 7 Drawfaig Sheets 
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MULTIUSER/MULTI POINTING DEVICE 
GRAPHICAL USER INTERFACE SYSTEM 

FIELD OF THE INVENTION 

The present iovention relates to &e field of event-driven 
grs^diical user interface systems, and mere paiticularly to a 
system for modifying a gr^hical user intciface response to 
user generated events in order to allow inultq)le user iiqmts 
to act in concert 

BACKGROUND OF THE INVENTION 

A number of gr^ihical usei interface (GUI) systems are 
available. These include Micros(tft Windows, IBM OS/2, 
Sod Microsystems Solaris, and Ai^e Conq)uter Macintosh 
Operating System. These systems have a pointing device 
iiq^Mt associated with a display, with a screen cursor defining 
an active position. The cursor typically is a gr^)hic object 
defining an active i»xeU region or a position^ tt» location of 
whidi on the screen is varied by man^)ulation of the 
pointing device. The screen image may Include a number of 
display objects, including puU-down menu bars, dialog 
boxes, windows, and icons. Superimposing the 00:5(7 over 
a defined screen object allows that object to be selected, with 
a resulting function activated. 

These GUI systems may allow a number of programs to 
be available, although not all allow true simultaneous inter- 
action. 

These interfaces may be event-driven, ie., change state 
based on defined occurrences, such as user input, timer 
OQ^t, or intenupts, or may continuously monitor inputs to 
determine thdr status. 

Known user pointing inputs include the mouse, traddball, 
joystick, light pen, touchpad, tablet and toudiscreen. The 
touchscreen, touchpad, li^t pen and tablet provide absc^ute 
positional coordinate il^xlts, while the mouse, trackball and 
joysdck provide relative motion iiqiuts. A number of differ- 
ent types of toucfascreens are available, including surface 
acousdc wave, resistive, aqpadtive, infrared, and force- 
sensitive types. 

In commonly known acoustic touchscreen position 
sensors, a transducer emits an acoustic pulse, which is 
spread as a wave across a touchscreen, and received by the 
same or different transducer. The wave has a diaracteristic 
time delay for each incremental distance along a common 
path, so that the position along a coardioate axis may be 
determined by a time-based analysis of the received pulse. 
Systems are provided for at least two orthogonal axes. See, 
U.S. Pat Nos. 4,642,423, 4,644,100, 4,645,870, 4,700,176, 
4,746,9 14 and 4,791/1 16, U.S. Pat No. Re, 33,151, U.S. Pat 
Nos. 5^60^21, 5,234,148, 5329,070, 5,177327, 5,162,618 
and 5,072,427, incoxporated herein by reference. See also, 
U.S. Pat No. 3,673327; Knowles, T. J., "46.6: A Pressure- 
Responsive Tbuch-input Device", SID 92 Digest, (1992) pp. 
920-923 ; Christensen, R. and Masters, T. , "^Guided Acoustic 
Wave: Newest Wave in Touch Technology*', ECN (January 
1995), pp. 13 ct scq. 

Con^Miter mice are also known. In commonly known 
devices, a hand held device having a con^uter connection 
cable is provided. The device and cable configuration 
vaguely resembles a mouse, hence the name. In a mechani- 
cal mouse, a spherical ball is provided which grips a 
reference surface and tracks a relative movement Internal to 
die OKxise, the rotation of the ball along two oithogonal axes 
is measured by wheels which press against die ball, gripping 



>4,150 

2 

along a respective rotational axis and sluing along a 
respective non-rotational axis. Thus, cadi wheel resolves an 
orthogonal conqranent of rotation. A mechanical or optical 
sensor is provided to determine the amount of rotation of 

9 each wheel, which is transmitted by the cable to the com- 
puter as a relative movement along each axis. Cordless 
designs, as well as designs with no moving parts are also 
known. Trackballs enqdoy an inverted configuration, with 
the hand or fingers of the user serving as the reference 

10 surface. 

Multi-screen disf^ay driver systems are known. These 
systems allow multiple viewing ''pons** on a single GUI 
operating system workspace, such as might be employed in 
process control or mnltimedia ^}plications, where display 
screens may be dedicated to particular infonnatioo or tasks. 
The virtual image or ''desktop** is enlarged across multiple 
displays, such as with a video walL These systems, however, 
do not include (ffovision for processing simultaneous inputs 
bom a number of user inputs to control the GUI 

^ Known multiple-display systems arc made by STB 
Systems, Richardson, Tex. including the MVP-4X Ad^>tcr. 
ColoigTaphics Conmiunications (Atlanta, Ga.)n^ufactures 
a number of mult^le VGA adulters, including PCMCIA 
Voyager VGA, TWin Turbo Acoelmtor, Super Warp, Warp 

" 4, and POS Video Adapter, Dual VGA Flap Panel, Pto 
Lightening Scries, which allow a Microsoft Windows, Win- 
dows NT or OS/2 based conqwter to have more than one 
active display. Multi-display adapter cards are also available 
from Easy Systems (Belgium), and Micro Deutsdiland 

^ (Gcnnany). 

U.S. Pat No. 5,437,014 (Busboom, et at), incoipmted 
herein by reference, relates to a system having nutitiple 
terminal devices dependent firom a main processor, support- 
ing mouse events, and executing multiple api^ication pro- 
grams. The main processor separates the activity of the 
dependent workstations, and tiierefore has mult^le work- 
spaces. Each mouse event is therefore associated with a 
workstation task. 

40 VS. Pat No. 5,442376 (Tannenbaum ct al.), incorporated 
herein by reference, relates to a multi-tasking graphical 
environment suf^xsting a plurality of ii^Mits. These ii^ts 
are not symntctrical, although a consistent interface is pro- 
vided. Therefore, each input source is particularly disttn- 

45 guished during processing. 

U.S. Pat No. 5,442,788 (Bier), incorporated herein by 
reference, relates to a multi-user multi-device system in 
which a plurality of us^ can control a single screen. Eac^i 
user is provided with one or mcHc input devices, which can 

50 be used to control one or more plications. At any time, the 
system produces a consistent view of all applications on die 
single screen. The input trom eadi user produces a response 
customized to the preferences of ttiat user, and events caused 
by these inputs are identified by source. Inputs to tlie system 

55 initiate the buHding of Event Records that are queued and 
then directed to specific applications. The software manages 
input streams firom multiple devices at once, keqnng track 
of which input comes from which user. Except where user 
actions conflict, the system allows all activities to proceed at 

60 once; where diere is a conflict, updates wait for a pause. 

SUMMARY AND OBJECTS OF THE 
INVENTION 

The present invention applies a muUqtle screen display 
65 driver system, with a oone^nding plurality of user inputs, 
to provide a multi-user environment on di^laced portions of 
a large virtual image space. Thus, a single conqxiting system 
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having an event-driven GUI qicrating system may receive keyfooard input events are not oonsidered in the analysis. If 

input from the plurality of input devices and control the a new button down event or new button up event occurs, 

di^lay or displays, providing each user ^ipiarent exclusive then a counter is incremented or decremented, respectively, 

and unintenupted control over a dedicated region or inter- Otherwise, the counter remains unchanged, 
active control over a shared regions object 3 1\vo modes of operation are defined. In a first mode, a 

Because ttte GUI operating systems used in conjunction plurality of usm are intended to operate indqxsdcntly, 

with the present invention are oammonly event-dnven, with without interference. In a scooikI mode, one or more uwr 

a single event queue and a single viitual workspace, an event inputs are intended to operate cooperatively, 

associated *ith one user input exdusivdy apjEoptiatcs the ih the first mode, if a new toudi occurs, i.e., if the touch 
processor during cvemprDCttsing.TTius, an event is pla^ lo eountex has been incremented, a button dicls, e.g., cursor 

in a queue, ^d each event is serviced by a service routine move followed by a *n>utton down" followed by ^Tnitton up" 

generally ^ the order of occurrence or priority, TTje present sequence is generated flus sequence filters the event stream 

invention does not alter the event proccsMg by including an to only assert a new iMtton down" event on new touches, 

h^t deWoc identifiia- with the pohiting device event, here- if no new toudi is detected, then the system takes no action 

Inafter referred to as the mouse event, to the queue, and ^ ^^^^ continues processing the event queue. In 

therefore does not require substantial modifications to exist- this mode, drag and douWe-dick operations arc not sup- 

mg operating systems or appUcations. Therefore, the input ported. On eadj new touch, the cursor position is control!^ 

device driver sj^tem nmst process the moose events in by the associated pointing device. Therefore, an absdute 

^"^^ transmh tbcm to the operating system position pointing device, such as a touchscreen, causes the 

andq)plicationiffograms.Thiscontcxtmaybe,far cjiample, 20 ^^^^^ ^ ^ location of each touch 

^Ln^l^^^^S!;^!?^.'^^ ^^"^ m the second mode, the critical question is whethrx the 

events, b^Joato referred to as niouse taton depression ^ dc<ieme«tcd to zero, as tiiis distinguishes 

r^^fatSu?^^ whena^etcjK* 

uumu« Y«iwuMin«v ucvi«5. of a multitude of touches was released and at least one touch 
Smcc tiicsc opcratmg systans are designed for a single ^ is still fwesent In general, the second mode aUows coop- 
simultaneous user, tfie operating system docs not make eration between iiqMits by allowing a Tmnd-off" between 
provision fw the distinction between multqdc pointing two user inputs on the cursor position without generating a 
device iiiputs, and detects only single occorrenccs of mouse ^Initton up" eveait, ^ch normally terminates a processing 
button events. Tberefne, prior art systems subject to a sequence. Thus, if there are multiple touches registered, the 
plurality of simultaneously active user inpat devices were ^ system continually pit>oesses tiie event queue, but docs not 
unable to resolve events created by different user inputs jhoccss a touch-release as a 1>utt<Hi up" event until the last 
acting in that same virtual di^y space, causing ctmfimon touch is released. This aUows the two or more user inputs to 
and mtcif ereocc. The prcscirt invention thcrcfOTC provides cotopetc for system attention, with none actuaUy gaining fiiU 
an input device interface system for determining a change in control, until it is the last or only iiqwt Thus, a first user 
system status as a result of a processed event frOTi a usa input be used to select an object within toe virtual 
input, thoreby allowing prooessii^ of that event in context, diqjlay apace, which may then be handed off to a second 
and not disturbing tiie i^ooessing <rf other unrelated events, user input on a secoiKi virtual display space for con^etion 
so that event processing may proceed without apparent of an operation. When the toudi counter is decremented to 
intcrfaence. ^ jgg^^ a ♦nwtton iq)" event is finally generated. If the touch 
Hie pfcsent invention prcferabiy does not divide or seg- counter is not decremeiUed to zero, the user inputs are 
maA the virtual display space into separate user areas^ nor processed as "drag" events, even if a new touch is added or 
particularly distinguish pointing inputs from different an old touch removed. In the second mode, drag and double 
sources. Therefore, pointing hiputs from different sources dick operations are siqiported. 

^jpcar symmetric. Ralher, two different user inputs may ^ In die first mode, the system software is designed so that 

mtCTct when desired, e.g., to indicate the source and each sequence of events for each user may be interrupted 

dcstinatioa of a drag operation. The present invention and that significant individual events are buffered. Thus, 

indudes a *inouse button d^ression" countcar to define a each user ^>pears to have control over the system, because 

status<rftiicsyst€mjHiOTtoprocessingancventandtoallow the inteoruptions are m<Hnentary and transparent, for suffi- 

detcrminatiwiof a diangc in status as a result of processing ^ dentiy powerful computer systems, 

the event TOs aspe^ of opermtion is important because user Preferably, in order to avoid interference between users in 

"Tl'^.^"^ the first mode, eadi user is provided witii access to a 

gcrcd by "button down and "button up" status, particular ^w^cation, and is rt^ded access to the oper- 

The present system therefore allows discrimination oE ating system as a whole. This is implemented by providing 

status changes relating to a plurality of user iiQHits wi&out 55 eadi user witii a GUI display space which does not indude 

altering the stnicture of die operaiing system. Since ttiis system level functionality and limiting die ability to csc^ 

functiooality may be ccsnpattmentalized as an intenupt the boundaries <^ die display space. 

scrWcc routine, i»ocesskg new user Inputs, a prcfared where absolute position pointing devices, such as 

incrementation is a software-based device drivo- for a touchscrecns, light pens, pressure sensitive screens, are 

^I^Hitrng system, whidi re^nds to an intern^rt sffvioe «, am)loyed, die device driver for eadi sudi pdnting device 

provides the operating system with a painting coordinate. 

Upon receipt of a new input occurrence, an event is Therefore, the pdnting input is unambiguous. WUh these 
defined and placed in a queue for processittg. This event types of pointing devices, no visible cursor need be pre- 
defines die type of device (e.g., pointing device), location sentcd to die user, since die position of touch identifies die 
(c-g., X, y coordinates) and stahis (e.g., button down, button w cursor position. In diis case, die rqrid movement of die 
iq>). It is presumed that all of die relevant user ii^Hit devices syston cursor position is invisible to die user, and need not 
have die same device type in the event queue, ie., that be addressed. 
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Id the case of other poiatiiig devices, such as toucbpads, inteiface; a plurality of user input devices, each user input 

mice, and trackballs, it is preferable to present a visible device being associated with one of said displays, and 

cursor as feedback for the of die ctrsar, so the user as to Ihe producing an input to said user input interface; said comr 

cursor position. In this case, since each pointing device putcr system executing an event-driven control program 

event generates an intemipt, updating the cursor positions, a 5 including a QUI having a unified virtual display space, each 

visitde cursor would flicker between the various pointing of said input devices jHodudng events; being for aimultar 

device position, causing an arbitration problem among the neously receiving user ii^ts firom said plurality of usa 

various pointing devices for control cf the cursor. Therefore, Input devices, displaying information rd^g to said user 

it is preferable in this case to deactivate display cf the system inputs on said plurality of displays, said gr^>hic image 

cursor, and Instead captm the pointing position of eadi lO presented on each of said plurality of displays being at least 

pointing iiqmt device and display a ffaptdc object on the a pcation of said unified virtual display space, wherein at 

saeen associated with the position of each pointing device. least two of said plurality of user input devices may used 

Therefm, multiple user curses will appear in die virtual simultaneously to input GUI commands, with each user 

display space, one for each pcinting input, yet the operating input having near real-time response to user input and each 

system stiU recognizes a single cursor positioa The capdir- 15 user ii^t device having iq)parent continuous contrd over 

ing of pointing device input and display of such graphic said graphic image of said associated display, 

objects in the virtual display space at an aj^xxfiriate actual The disj^y image may also be an image pcesented on 

display position is within the cidinary sJdll in die art multiple display devices simultaneously throug^i die use of 

In the case of relative positioa pointing input devices, video splitters, whidi allow the presentation of the same 

such as mice and trackballs, a furdier enhancement is ^ image on multiple displays for use by one or more users, 

preferred. Normally, these devices generate cursor relative According to one embodiment of the hivention, a plurality 

position movement data. This is appropriate if die starting of transparent touchscrcens, siq>erimposed on display moni- 

position ci the curses is defined by die past histny of tors are iHt>vided, each allowing a user to touch the 

movements of a particular pointing device. However, where toudisoeen, resulting in absolute position events, allowing 

a plurality of pointing devices iDdq)endeotly reposition the 25 an event to be generated t>y a simple touch or drag across die 

curs<v, die starting position may be inappropriate. Therefore, screen. Each display monitor may display die same or a 

a driver is provided to translate the relative pointing device different image. 

output to an absolute coordinate system, aj^qpriate for die jhe even^driven control program is preferably a GUI 

actual display device widi which it is used and die virtual system selected ftom die group consisting of Wndows, 

display space. Advantageously, diis function may be com- ^ Endows for Wwkgroiqw, Windows NT, Windows 95, OS/2 

bincd with the graphic user cmsor generating system Macintosh operathig systems. The displays are prefer- 

dcscribed above, which will define an absolute user cursor ably color raster cadiodc ray tube (CRT) dis^ilays, but may 

position limited widiin the virtual display space. Thus, also be liquid crystal, electroluminescent, vacuum 

instead of transmitting messages defining relative move- fluorescent, light emitting diode, cold cadiode or other 

ments of the cursor position, the driver system defines display types. 

coordinate messages based on the pa^ history of movement -j^ jj, preferably comprise transparent 

of the particular pointmg device. This allows die use of touchsaeens superimposed on said associated display, 

various pointing devices. ^j^^^ ^ acoustic wave, resistive, capaciiive, 

It is noted diat the input device driver docs not alter die inftared, force-sensitive or odiex known types. Of course, 

operation of die <^)erating system, which is still essentially ^ other types of p(»nting devices may be employed, 

a single user type, widi a single system cursor location. system jccfcrably includes at least two display 

Acc<7ding to die present invention, user inputs are pro- devices and two user input devices. The di^lay driver may 

ocsscd symmcSiically based on pointing location, widiout ^ ^ integrated multi-display ad^)ter or video splitter, 

si g nifi ca n t distinction as to input source. The present system ^^ff^^ igp^i interface may be a multi-input device, a 

allows diis single user, multi-tasking (^xrating system to daisy-chained inpvit device » separate devices. 

tppcar as a multiuser opaating system. According to anodicraq>ect<rf die invention, die multiple 

Some display driver devices support hardware cursor display GUI system is provided wherein said event-driven 

display, and thereftxe may be used to show a user cursor control program including a QUI having a unified virtual 

somewhat independent of die operating system Odierwise, ^ dis{^y space, produces a plurality of display windows in 

the cursor position disjday and ooordinate conversion oper- said unified virtual display space, only one of said display 

ate as a device driver or application program. windows bang associated widi processed events at any 

Accmding to the present invention, a normally single user instant, an input to one of said input devices producing an 

interface system, confonning to certain rules, is made into a event associated with a window in said portion of said 

multiple user interface by providing a plurality of input 55 unified vutual display space of said associated display 

devices, a virtual display screen enconq»ssing a number of device, which is then {»ocessed. 

display windows, having a phirality of real display ports, The system may be in:q>lemented widi comparatively 

wherein each display window is associated with an area of standard hardware and operating systems, such as a proces- 

said virtual display screen, an input from one of the ii^xit s<x executing the 80XS6 instruction group processors. Of 

devices associated with an area of said virtual di^lay screen ^ course, other architectures, such as PowerPC, MIPS, 

being associated with the associated display window, SPARC,AlFha,GrodicrCISCo(rRlSCmicroprocessOTsmay 

thcr^ transferring control to that window. be used. A plurality VGA, Supct VGA or Windows 

It is therefore an object of the present invention to provide accelerated display drive systems, integrated on a single 

a multiple display GUI syston cosiqnising a computer board or on multiple boards, may be used. Video splitters 

system having a processor, a disfday ouQHit interface and a 65 may be used to present an image on multii^e displays, 

user input interface; a plurality of displays, each presenting For a full understanding of the present invention, refer- 

a graphic image and bdng responsive to said display ou^ut ence should now be made to the following detailed descrq>- 
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tioD of the preferred embodiments d Ibe inveotioD as FIG. 3 shows the system generally aocordiDg to FIG. 1, 

illustrated in the accompanying drawings. having a multiple output SVGA display driver 14, 15, STB 

BRIEF DESOUFnON OF TfffinRAW^ """^ MVP-2X, driving SVGA monitors 4, 12, having 

BRIEF DESCRimON OF THE DRAWINGS touchscrecns 5. 13. A single virtual diq>lay space is provided 

The pfefened embodiments of die invention will t>e 3 for the qjplications.^^ch may span the various monitm 4, 

shown by way of drawings of the Figures, in which: 12. Thus, portioos of qi(rfication A 1#, If', may be displayed 

FIG. I is a scmistiiematic view cf a single user system onmoaitors4,12,andlikcwise,portionsof ^UcationB 11, 

running multiple windowed applications; ^ displayed on mooitors 12, 4. Toucfascreens 5, 13 

FK}. 2 is a semischematic view cf a single user system may be used simultaneously, 

ninniog multiple windowed applications with a video ^ shows the system generally accofding to FIG. 1, 

splitter, displaying the same image on mult^le displays; having a multq)le output SVGA display driver 24. 25, 26, 27, 

FIG. 3 is a semisduanatic view of a system running ^ MVP-4X driving SVGAmooitors 16, 18, 20, 

multiple windowed applications in a single virtual desktop, ^ ^ touchsdcens 17, W, 21, 23, respectively. A 

with a mult^KJrt video display adi^^ter and a plurality of single virtual display gwcc is provided for the applications, 

(jifipjays- 1^ AsshowninFIG,4,aainglei^plicationwindowAi,Aa,A3. 

HG. 4 is a scmisdiemalic view of a system running ^'j!''^''^^'^?'^^ 

multiple windowed applications tiled on Zuipic display exdusive access to 

adapt^ in a single Xd desktop; ^ ^ multiple oopi^ or instaces of Ih^ 

e . ~r *J^L* J ^ Whcahon B is a different appUcation. These appli- 

Sisaflowchartfbrafirstmodeof opaation 2o cations are tiled across die virtualdcsktcp so they ^ 

accordmg to die present mvention; maximized on each display. Toucfascreens 16, 18, 20, 22 

HG. 6 IS a flow diart for a second mode of <^)cration may be simultaneously used, with no intcrfacnce between 

according to the presoit invention; various uscn. 

HG. 7 is a flow chart combining first and second modes Multicopies or instances of a point of sale program, or 

of operation acceding to the present invention; 23 ^^rcLt pr<^rams, arc run, with separate windows in die 

FIG. 8 shows four visible user cursois and an invisible virtual display space. Each of the windows is presented on 

system cursor on a virtual desktop spread across four a separate monitoL The touchscreen of each monitor tfaer&- 

mooiton, as shown in FIG. 4; and fore overlays a separate window. A touch on one of die 

FIG. 9 shows a flow chart for processing a relative touchscreen surfaces over a window triggers an event in die 

position pointing device iiqHit ^ operating system wfaidi causes diat window to be "active**, 

rvf^^wT rv«»^»*T^^^T 6^ « switch context Each POS iqjplicatioa may 

PREFERRED EMBODMEOTS eadi of these screen objects^ provided in a mamier such 

The preferred embodiments of the iR^esent invention will diatany sequence of inputs may be intcmiptcd,i.e., an open 

now be described widi lef crenoe to die Figures. Identical dialog box does not lock die inputs preventing switching of 

elements in die various figures of the present invention are contexts. 

designated with the same reference numerals. Because diese GUI operating systems are essentially 

A systemis provided having a typical Windows, Windows designed for use at any one time by a single user, the general 

fot Workgroups, Windows 95, or OS/2 configuratioii, such 40 programming model provides for but a single master 

as an 80486 DX2-66 or Pentium 66 processor, 16 MBytes of program, Le., Ftogram Manager in ^Mndows, Finder in 

memory, and a 500-1080 MByte hard drive. A phnality <tf Macintosh, etc. Therefore, it is preened that this high level 

SVGA touchscreen monitors, having Elo toudiscreens (Oo functionality be hidden or inaccessible from any nc^mal or 

TouchSystems Inc., Oak Ridge, Tenn.), are interfaced aberrant user input, in <Hder to avoid disruption of system 

dirough an STB MVF-4Xdis]day adulter and a plurality of 45 actMiy fca: all usees. In addition, ^ddle generally GUIs may 

RS-232 potts, using Elo **MonitorMoase" s<^are driver he used as a primary input to interact widi the systems, often 

for Windows, ^Ondows NT, OS/2, or Macintosh in accor- functionality may also be accessed from a keyboard, 

dance with the present invention* These drivers allow typical Therefcae, in a prefeored embodiment, a keyboard is not 

^^dows user input from ttie touchscreen ii^mts, sudi as available for normal users. Alternately, a keyboard ii^Nit is 

dick, double-dick, move and resize window, drag and dn^, ^ filtered to prevent unintended disruption of other users and 

scroll and pull-down menu selection. This system also the input data associated with a corresponding pointing 

allows DOS sessions to run widi touchscreen ii^Hit, as well device input 

As known in the prior art, a system, as shown in FIG. 1, Noonally, drag operations are unintcnuptablc from a 

executes a plurality of ^){dications, each occupying a win- system stand^int, but this need not be tite case, as with an 

dow on a ntonitor. This system executes Windows for S5 event driver architecture, a *Mease** event may be dcter- 

Wodcgroups operating system and MS-DOS 6J22. An Intel mined without requiring continuity of input of "depress** 

80486 DX2/66 processor 1 is provided, having a standard events. 

con?>lcmcnt of periphcrals.anclnding a mouse 2 and a Therefore, it is a fundamental dement of the present 

keyboard 3 An SVGA monitor 4 is provided, having an Elo invention ttut, in a unified event-driven envirwiment, an 

touchscreen 5. During use of the touchscreen 5, die mouse 60 event server Is provided haWng suflident bandwidth to 

2 is inactivated. Two windowed plications, 6, 7 are process event streams from a plurality <rf input devices in 

diq>layed on the monitor 4. quasi-real-time, allowing event streams from different 

FIG. 2 shows the system generally according to FIG. 1, sources to be processed generally in oidcr of occurrence. It 

having a video splitter 9, diqilaying windowed plications is noted diat the system is not necessarily a xval-time 

6. 7 on monitors 4, 4* having touchscreens 5, 5*. The system 65 processor, and events fpcnn differing sources need not be 

according to the fncsent invention allows simultaneous use processed in strict order of time occurrence. Events from the 

of touchscrecns on both monitars simultaneously. same source should be jKocessed in orxUr, although they 
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may be batched. Batching of events may be e£Scient where devices may appssa to simultaneously drag the same screen 

a series of events coiiesponds to a command input It is object fluidLy as the object quiddy moves back and forth to 

noted that cursor movement events should be prcccsscd in as the position defined by each int^laced inteui^ without 

dose to real-time as possible* and it is possible to execute mutual inteiference. 

these commands out of order, i.e.* by placing other types of 5 Alternate to allowing the standard system supervisory 

conunands in a qurae. Thus, events nuy be processed in at program to control switching among multiple windows, 

least two phases, a detennination of type and an execution whidi may be multiple instances of the same application, a 

phase. specialized qiplicatioa program may be provided which 

Thus, using a touchscreen^ a touch or rdease in an area occupies the entire virtual screen area, and dius all inputs are 

causes an event whidi indudes the time (or sequence) of the fffocesscd through the specialized qjplicatioQ program. TTie 

touch or release as well as the location of the toudi or specialized application program, in turn, may have daughter 

release. If the touch is on a screen area wWdi is not windows which may optionafly confonn to standard pro- 

controUed by a window, then the event is processed by the gram objects of the system or be totally custonL Thiw, in the 

supervisory program, and as a result may produce no visible case of amuW-sciccn application prcjpam, no modifi^ons 

eff^ If, on the other hand, the touch ir rdease h are nec^sary to event pjooessmg at the opcratmg system or 

screenareacontrolledbyan<^window,itisprocesscdby supcrvistHy sy^ levd. , ^ , 

acontrolprogramforthatwindow.ffthetoudiarrdeaseis The method according to the P«^5, invention is 

on an area overlying an inactive window, in certain described in die flow charts presented in FIGS. 5, 6 and 7. 

drcumstances, the supendsory program may intercept the PIG, 5 shows a first, multi-user, point mode of operation^ 

event to make that window active, without further process- ^ ^^"^^ * ^'^^ ^^^"^ "'^^ ™^ 

ing. This is oormaUy a **safety- feature based on the pre- operation. FIG. 7 shows a system optionaUy having both the 

sumption diat transferring control brtween windows is not a first, point mode and die second, drag mode of operation, 

normal mode of operation and that therefore a delay should Each touchscreen input produces an interrupt, which is 

be haposed to prevent inadvertent execution by a program cq)tured by an interrupt service routine 101. Touch data, 

controlling a window which was recenay inactive. Accord- ^5 induding coordinates (X, Y) and toudiing information 102. 

ing to the {wesent invention, this safety feature is diminated The touch data is then tested to see if a touching state has 

or altered in most cases, so that die toudi or release causes changed 11(3, c.g., whether there is a new touch or rdease. 

a focus change to the controlling program and a command If there is a change 103, and a touch is present 104, a counter 

widiin that program ot window. Hius, an active window and is incremented 105. If there i s a change 103, and a toudi is 

an inactive window each qppear to respond identically to a 3^ not present 104, a counter is deacmented 106. 

touch input event Thus, since the active and inactive win- In FIG. 7, two modes of qjcration arc provided, which arc 

dows ^Jpear to <^ate identically, fiirttia discussion will selectivdy executed 107. In the point mode, the critical 

focus on the inactive window, with active windows requiring dedsion is whether the counter is incremented in the initial 

possibly fewer steps. Prior art systems may provide a visual processing step, Lc., whether the new counter value is 

change t)etween active and inactive windows, however, 33 greater than the initial counter value 108. indicating a new 

according to die present invention, it may be piefaable to touch. Is the new value is greater, a series of event messages 

have an identical appearance to give the effect of continuous are queued, a move, button up and button down messages 

user control ova: the system, when in fact the resources arc 109, and the old touch counter is updated 110. The intern^ 

being shared. soiice routine is then terminated 111. In this mode, the 

In a multiuser intofoce system, in order to allow a user to 40 various inputs are indq>endcnt and do not interfm with 

receive feedback &om a touch event instantaneously, and to each otbo", as only a new toudi event generates messages, 

p-event interspersing of events from otfier user iiqiuts, In the drag mode, the critical decision are whether the 

interrupts caused tyy other users are blocked untH die a touch counter transitions to 112 or from zero 113. If the 

mouse move, dick down and didc up events are queued In touch counter is one and the pricB' touch counter value is 

the interrupt service routine. While this may dday slightly 45 zero, then move and button down messages are queued 115. 

iiiq)erceptibiy to die user the initial response of the system Otherwise, a move witii button down message (drag) 114 is 

to a user iiqwt, it allows uninteinq)ted response during an generated. Tf the toudi coimtcr is now zero, a button up 

input message is goiaated 116, The touch counter is updated 110. 

In a single user system, a window may be dragged The interrupt service routine is then terminated 111. This 

between display units by touching the window with one 50 allows die first touch to produce the button down message, 

band while simultaneously touching the intended location toid the last rdease to g^erate a button up message. Thus, 

widi the other hand, on a different display unit, and subse- in the drag mode, various ii^ts are interactive, 

quentty lifting the touch on the first display, whereby the FIG. 8 shows four visible user cursors 30, 31, 32, 33 and 

window will move to the location on the second display unit an invisible standard system cursor 34 on a virtual desktop 

When the drag event is initiated, the first touch event is 35 spread across four mooitors, 16, IS, 20, 22 as shown in FIG. 
detected, with the series of coordinates used to move the 4. Each of the visible user curs(H5 are assodated widi 
object So long as die touch is not interrupted, the window relative pointing devices 2, 35, 36, 37, rcspcctivdy. Accord- 
remains in a drag state. The intemqyts caused by odier ing to this system, position of the user cursor for each of the 
simultaneous events are handled in queued sequence. This is four pointing devices syston. The cursor generation func- 
espedally true wtieretouchscreens or input devices fcH* each 60 tions are maintained by a device driva or qjfdication 
di^lay have the same interrupt rate. The system hardware program. The device driver or j^lication program main- 
has suffident t)aDdwiddi such diat each input device has a tains variables LastUscrX., LastUsaY„, storing absolute 
fliud cursor movement and apparent fiill control over the coordinates for each user cursor. Other information regard- 
system. Thus, in contrast to standard-type systems, input ing tiic history of user iiq>ut may also be stcH^ such as 
events continue to be recdved and processed even as one 65 vdodty and gestures. 

input device causes a stream (^events, such as by continual When a relative motion is received from a pointing 

touch mouscbutton de{ve8$i<Mi. Thus, mult^e input device, eg., 2, 35, 36 ot 37, die associated user cursor 



11/26/2002, EAST Version: 1.03.0002 



5,694,150 



11 



12 



position is updated using known methods, generally based 
on me LastUser?^, LastUserY„, and opdonally other user 
input informatioD. Hie visOde user cursor, e.g., 30, 31, 32 or 
33, however, does not always correspond directly to relative 
motion undo* the foUowing dtrcumstanoes. The user cursor $ 
is limited to movcxncot within the associated virtual display 
window. The system cursor 34, which can be located any- 
wtiere across the virtual desktq>, is not necessarily iiu>v^ 
as a resuU of pointing device input The system cursor 34 
may be kept kvisiUe, oi having different graphic charac- 
teristics. The system curses 34 is updated, and system mouse 
events geaerated, according to the methods of FIGS. 5, 6 and 
7. 

Hie operating system is fsovided with absoktte pointing 
device positions, even \i^en relative motion-type devices 
are used. Therefore, the mettiod shown in FIG. 9 prcpro- 
cesses pointing device inputs from rdalive motion-type 
devices, and is generally not employed with absolute 
position-type devices. One relative motion-type pointing 
device may optionally remain associated with the system ^ 
cursor, as in known systems, and have access to all nonnal 
operating system pointing device input functions. 

Where a relative position pointing device is employed, 
such as a mouse 2, prior to processing the pointing events 
accOTding to the procedures of FIGS. 5, 6 and 7, a prcpro- 25 
cessing px)cedure according to FIG. 9 is cn^iloyed. This 
preprocessing procedure performs two functions. First, a 
user cursor is generated for the pointing ii^wt device at a 
position related to the past history of use of the pointing 
device. Second, &e message passed to the cpmting system 3^ 
related to the pointing device iapat is translated to an 
absolute coordinate positioa 

An interrupt 120 is generated by pointing device use. The 
relative motion of the painting device is determined 121, and 
processed in conjunction with stored infoonation relating to 35 
die past Mstoay of use of a particular pointing device, e.g., 
a set of stored coordinates. This stosed information may also 
be more coinplcx, for example including velocity data for 
use in confuting ballistic cursor movements, or gestures, as 
in known systems. A user absolute coordinate position is 40 
thus c<Hiq>uted 122. Hie user curm movement is limited 
within the virtual di^y space 123. The stored infonnadon 
is updated 124. The user cursor position is tbcn displayed 
125 on the di^lay device associated with the particular 
pointing device. 45 

Hie pointing deWoe message is then pnqiared and trans- 
mitted as absolute coordinate X, Y, refened to as touch data 
in step 102 of FEOS. 5, 6 and 7, and interrupt service 
routine terminated 127. 

There has thus been shown and described novel aspects of 50 
multi-input systems, which fulfill all the objects and advan- 
tages sought therefor. Many changes, modifications, 
variations, combinations, subccmfainations and other uses 
and i^licatioos of the subject invention will, however, 
become ^iparcat to those skilled in the art after considering 55 
this specification and the acccn^Muiying drawings which 
disclose the preferred embodiments tfaerec^. AU such 
changes, modifications, variations and otiia uses and pli- 
cations wHdi do not depart from the spirit and scope of the 
invention are deemed to be covered by the invention, whldi 60 
is to be limited only by the claims which follow. 

What is dainoed is: 

1. In a computer system including a processor, and 
employing a graphic user interface having a virtual display 
space, executing an event-driven control program, wi± a 65 
unified user pointing iiqiut event stream processing system, 
the improvement conqirising: 



the control program receiving at lea^ first and second user 
iapats representing different locations within said vir- 
tual di^lay space, eadi of said user ii^Hits t>eing 
associated with the production of events of said control 
program and being processed without distinction 
t)etweea said first user iiq>ut and said second uscrii^t; 
and 

said control program having an operational mode for 
processing a series of events related to a first user input 
without apparent functional intcrfcxence with a concur- 
rent series of events related to a second user ii^Hit 

2. Hie counter system according to claim 1, whcrdn 
said event-driven control program is a graphical user inter- 
face system selected from the group consisting of Windows, 
Windows for Wockgroi^, Windows NT, Endows 95 , OS/2 
and Macintosh operating system. 

3. The con^uter system according to daim 1, further 
con^Hising a plurality of displays, each displaying a portion 
of die virtual display space. 

4. The coE^Miter system according to claim 3, vt^imin at 
said plurality of displays each display a differing portion of 
the viitual display space. 

5. The computer system according to claim 1, vtlicrein at 
least one of said user iiq>uts conqxises al>sdute position 
coordinates. 

6. The con^ter system accoaxling to daim 5, vvfacrctn 
said user input receives absolute position coordinates from 
a toucfa-sensitrve device superin^>osed on said display, so 
that a touch directed at a corresponding display location of 
said touch sensitive device directs a cursor to said display 
location. 

7. The compter system according to claim 5, wherein 
said user input receives absolute position coordinates from 
a touchscreen. 

8. The computer system according to daim 1, conqirising 
at least two di^lay devices, eadi displaying a differing 
portion of said virtual display ^Mice, and at least two user 
input devices. 

9. The computer system according to daim 1, further 
conqjrising a plurality of display devices, each having an 
assodated user input, wherein each display and associated 
user iiqf»ut set are provided witti equivalent capabilities. 

10. The conq)uter system according to claim 1, wherein 
said virtual display space indudes s plurality of objects, 
eadi user ii^xit being associated with one of said objects. 

IL Hie conq>iiter system according to daim It, wherein 
said phnraJity of objeos comprises a plurality of application 
windows. 

12. The computer system according to daim It, wheidn 
said centred program si^pports q>plications having applica- 
tion windows, an sf^dication associated with an application 
window receiving events from the control program and 
further processing events relating to a user input associated 
with said applicadon window. 

13. The con^Niter system according to daim 1, further 
conqirising a plurality of displays, each displaying a differ- 
ing portion of said virtual display space, said control pro- 
gram responding to WIN16, WIN32 or OS/2 API syntax. 

14. The coii9>uter system according to claim 1, further 
con^Hising: 

a processor, having: 
a counter; 

a memory for storing a state oi said cmmter; 

means for Incrementing said counter upon an activate 
ii^Mit status from any of said inputs and decrement- 
ing said counter upon an inactivate input status from 
any of said iiiputs; 
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point mode means fcH* generating cursor move to said 
position, button down and button up messages in 
sequence if said counter is inaemented; and 

means far i^Klating said stored state d said counter. 

15. The conqHiter system according to claim 1, furttier 5 
con^Hislng: 

a processor* having: 
a counter; 

a memory storing a state of said counter, 
means for inooncnting said counter upcm an activate 
input status from any of said inputs and decrement- 
ing said counter upon an inactivate input status from 
any of said ii^xits; 
drag mode means for generating a button iq> message if 
said counter transitions to a value of zero, guierating 
a button down message if said counter transitions to 
a value of one and said stored initial state transitions 
to a value of zero, and otherwise maintaining a 
button down message state; and 
means for updating said stored state of said counter. ^ 

16. The conqwta system according to claim 1, further 
con4)dslng a system for converting an output of a relative 
pointing position user iiqmt to a display coordinate position, 
and a system f<s displaylQg a user cursor, based on said 
relative pcrinting position user input and indqsendent of said ^ 
control (nogram, on said display. 

17. An event-driven graphical user interface system hav- 
ing a virtual display space including a plurality of objects, 
comprising: ^ 

a user ii^t interface, being responsive to a i^urality of 
user inputs from identified user input devices, said user 
inputs each being associated with at least one object, 
and being associated with ttie production of events 
substantially undistinguished by a user input device 
identification; 

said interface system having a mode for processing a time 
scries of events idated to a first asei input relating to 
a first object and a concurrent time series of events 
related to a second user input relating to a second ^ 
object, without mutual inteifacnce. 

18. The event-driven gri9>hical user interface according to 
claim 17, further oonqirising a control program for control- 
ling said grq^cal user interface, having a single system 
cursor, a system for converting events generated by an 45 
output of a relative pointing position user input to a display 
coordinate position, and a system for displaying a user 
cursor, based on said relative pointing position user input 
and independent of said system cursor, on said gr^rfiical 
user interface. 50 

19. An event-driven graphical user interface system hav- 
ing a unified virtual display space including a plurality of 
objects, comprising: 

a user input interface producing events relating to a 
plurality of user iiqxits from at least one identified user 55 
irqxit device; 

said interface system concurrently processing a series of 
events substantially wltfiout reference to an user input 
device identiflcation, related to a first of said plurality 
of user inputs and a series of events related to a second 60 
of said plurality of user iBpvUs, without mutual intex- 
ference. 

20. The event-driven grq>hical user interface according to 
daim 19, wherein said intc^ace system has a single system 
cursor, further con^xising a system for converting events 65 
generated by an oatput of a relative pointing position usa 
input to a display coordinate position, and a system for 
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displaying a user cursor, based on said lelative pointing 
position user iiqiut and independent of said system cursor, on 
said graphical user interface. 

21. An input processor for a gcz^cal user interface 
system having a unified virtual di^lay space, comprising: 

an input for receiving inputs from a plurality of user 
porting device ii^ts, defining a position and an input 
status; and 

a processes, having: 
a counter; 

a memoiy for storing a state of said counter; 

means fos inCTcmcDting said counter upon an activate 
input status from any of said inputs and deaement- 
ing said counter upon an inactivate input status from 
any of said inputs; 

point mode means for generating asrscx noove to said 
position, button down and button up messages in 
sequence if said counter is incremented; and 

means for updating said stared state of said counter 

22. The system acccsding to daim 21, wherein said 
processor further comprises: 

means for selecting a mode of operation, sdecting 
between said point mode means; and 

dtag mode means fcr generating a button up message if 
said counter transition$ to a value of zero, genoating a 
button down message if said counter transitions to a 
value of one and said stored initial state transitions to 
a value of zero, and otherwise maintaining a button 
down message state. 

23. Tlie system according to claim 21, wherein said 
processor further conqxrises a control program having a 
system cursor, a system for converting said relative position 
pointing device ii^Hit to a display coordinate position, and a 
system for displaying a user cursor, based on said relative 
position pointii^ device input and ind^>endent of said 
system cursor, in said unified vhtual dig)lay space. 

24. An input processtv for a graphical user interface 
system having a unified virtual di^lay space, compdshig: 

an input for receiving inputs from a plurality of user 
pointing device iiqiuts, defining a position and an input 
status; and 

a processor, having: 
a counter; 

a n^moiy for storing a state of said counter; 

means for incrementing said counter upon an activate 
input status from any of said inputs and decrement- 
ing said counter upcm an inactivate input status from 
any of said inputs; 

drag mode means for generating a cursor move to said 
position and button down message in sequence if 
said counter transitions to one, generating a button 
up message if said counter transitions to a value of 
zero^ and otherwise maintaining a drag state with 
move with button down messages; and 

means for updating said st<Hed state of said counter 

25. Hie system according to daim 24, wherein said 
]»-ocessor further comprises: 

means for selecting a mode of cq>eration, selecting 
between said drag mode means and point mode; and 

means for generating "cursor move to said position**, 
*1>utton down** and "button up** messages in sequence 
if said counter is incrcznented. 

26. The system according to claim 24, wherein said 
processor further oon^irises a control program having a 
system cursor, a system for converting said relative position 
pointing device input to a di^lay coordinate position, and a 
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system for displaying a user ctirsor, based od said lelative 
position pointing deWoe input and independent of said 
system cursor, in said unified virtual display space. 

27. In a ffafbial user iotcrfux system having an event- 
driven control program with a unified user input pointing 5 
event stream without distinction of source and a gr^ihical 
user interface having a virtual display space containing 
objects, the improvement con^Hising: 

control pr<^ram jsocesslng events related to a plu- 
rality of user inputs representing different locations 
wi^^^l the virtual display space; 
the control program ha:ving one or more naodcs sdectcd 
from the group consisting of; 

(a) an operatiooal mode for permitting said plurality of 
user iiqnits interact to allow an object in the virtual 
display space at a first location specified by an initial 
user ii^ to be associated with a second location 
specified fay a concurrent and subsequently terminat- 
ing user input, the control program having a counter 
which increments on activity of any usa input and ^ 
decrements on cessation of activity of any user iiqmt, 
for determining an initial user input ccmdition, tem- 
poral concurrence of a plurality of user inputs, and a 
terminal user input condition; and 

(b) a second operating mode for processing a series of ^ 
events related to a first usa input without apparent 



functional inteiferaioe with a concurrent series of 
events related to a second user input 

28. The gn^>hical \iset interface system according to claim 
27, whmin the control program indodes both mode (a) and 
mode (b). 

29. The graphical user interface system according to claim 
27, wherein the control program includes mode (a). 

30. The graphical user intofaoe system according to daim 
27, wherein the control program includes mode (b). 

31. The graphical user intof aoe system according to daim 
30, wherein the control system cosqarises a counter, iocre- 
menting on occurrence of an event related to a user input and 
decrementing on oessatioo cf an eventrelated to a user iiqNit 

32. The event-diivefi graphical user inteif ace according to 
daim 19, wherein: 

each event is associated with a location and a touch status 
within said virtual display space independent of other 
ii^ts; 

said interface system having a mode for processing a 
series of events associated with a plurality of different 
iufNits, wherein an object at a location associated with 
an initial iqxit is transfemd to a location associated 
with a coexistent terminal input, compdsiag a counter 
for tracking a toudi status to determine a beginnittg of 
said initial input and an end of said terminal inpiit. 
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